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Abstract. Atomicity (or linearizability) is a commonly used consistency 
criterion for distributed services and objects. Although atomic object im- 
plementations are abundant, proving that algorithms achieve atomicity 
has turned out to be a challenging problem. In this paper, we initiate 
the study of systematic ways of verifying distributed implementations 
of atomic objects, beginning with read/write objects (registers). Our 
general approach is to replace the existing operational reasoning about 
events and partial orders with assertional reasoning about invariants and 
simulation relations. To this end, we define an abstract state machine 
that captures the atomicity property and prove correctness of the object 
implementations by establishing a simulation mapping between the im- 
plementation and the specification automata. We demonstrate the gen- 
erality of our specification by showing that it is implemented by three 
different read/write register constructions: the message-passing register 
emulation of Attiya, Bar-Noy and Dolev, its optimized version based on 
real time, and the shared memory register construction of Vitanyi and 
Awerbuch. In addition, we show that a simplified version of our specifi- 
cation is implemented by a general atomic object construction based on 
the Lamport’s replicated state machine algorithm. 


1 Introduction 


Many distributed and network-based services can be modeled as shared objects 
accessible to (possibly remote) clients through well-defined interfaces. Atomicity 
[16, 21] (also known as linearizability [10]) is a desirable property for such objects 
as it allows clients using the objects to perceive the operations that occur in each 
run as occurring atomically, in some sequential order. This perception makes it 
easier to understand the behavior of a system using distributed services, and so, 
simplifies the task of system design. 

Atomic services could be implemented simply on single server machines. How- 
ever, to achieve high availability in a distributed system and to tolerate failures, 
atomic services are typically implemented by distributed algorithms. Many dis- 
tributed algorithms have been proposed for implementing atomic objects; see, 
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for example, [17, 15, 36, 27, 33, 32, 10,35, 14,19, 18,22, 23,8]. These use a range 
of techniques to achieve the appearance of total ordering, for example, assign- 
ing timestamps and processing operations in timestamp order, or using quorum 
configurations. 


Although atomic object implementations are abundant, proving that algo- 
rithms achieve atomicity has turned out to be a challenging problem. Most exist- 
ing proofs for such algorithms are long, subtle, and difficult to understand and 
check. As evidence of the difficulty, we note that several published proofs for 
implementations of atomic shared read/write memory objects have later been 
shown to be incorrect. We believe that a fundamental reason for the difficulty of 
these proofs is their style: they are based on detailed, not-very-systematic, rea- 
soning about events and their ordering. Useful structure in such proofs is often 
provided by lemmas about partial orders of operations on objects, for example, 
Proposition 3 of [16] (for single-writer read/write objects) and Lemma 13.16 of 
[21] (for multi-writer read/write objects). These lemmas provide sufficient con- 
ditions for correctness of atomic read/write object implementations, based on 
a list of properties that a partial ordering of operations must satisfy. However, 
showing that these properties hold still requires detailed, ad hoc reasoning about 
events (see, e.g., [22, 23]). 


In this paper, we study systematic ways of verifying distributed implemen- 
tations of atomic objects, beginning with read/write objects (registers). Our 
general approach is to replace operational reasoning about events and partial 
orders with assertional reasoning about invariants and simulation relations. The 
assertional methods differ from the traditional operational arguments in two 
important ways. First, the system properties are stated precisely in terms of 
predicates over the system state components. Second, assertional proofs can be 
checked by examining individual state transitions of the algorithm without rea- 
soning about entire executions. As such they lend themselves to mechanization, 
i.e., the process of checking a proof can be carried out using interactive tools, 
such as theorem provers. 


Our approach to carrying out assertional atomicity proofs is first to define 
an abstract state machine that captures the atomicity property and then, prove 
correctness of the object implementations by establishing a simulation mapping 
between the implementation and the specification automata. The challenge is 
to find a specification automaton that is general enough to apply to many ex- 
isting implementations, and at the same time sufficiently close to the actual 
implementations to simplify the task of finding the mapping. One example of 
an atomicity specification that turned out to be too abstract for carrying out 
simulation proofs is the canonical atomic object automaton of Section 13.1.2 
of [21]. The canonical object automaton maintains a buffer used to store incom- 
ing client requests. Buffered requests can later be applied to the object state, 
and the generated responses are returned to their originators. Unfortunately, 
this specification, though simple, does not provide sufficient detail to allow for 
easy match with concrete implementations. 


We therefore, give more detailed specifications. Namely, we define an ab- 
stract state machine, which we call the Partial-Order Machine (PO-Machine), 
which records information about operations and their orders in its state. The 
PO-Machine expresses the common behavior of many existing atomic register 
implementations, in which client operation requests are gradually ordered rela- 
tive to other operation requests until all the necessary ordering constraints are 
achieved. The ordering constructed is, in the limit, guaranteed to be a partial 
order of the requested operations that satisfies sufficient conditions for showing 
atomicity. 

We use the PO-Machine as a formal specification for distributed algorithms 
that implement atomic memory. We show that it is implemented by three dif- 
ferent read/write register constructions: the message-passing emulation of At- 
tiya, Bar-Noy, and Dolev (ABD) [3] (extended to handle multiple writers as in 
[23]), an optimized version of ABD that takes advantage of synchronized clocks 
at writers [8], and the unbounded version of the shared memory construction 
of a multi-writer/multi-reader register from single-writer/single-reader registers 
of [36]. We also show that a slight modification of the PO-Machine, called the 
TO-Machine, can be used to prove atomicity of a general (i.e., not necessarily 
read/write) object implementation based on the replicated state machine proto- 
col of Lamport [15]. 

We specify the PO-Machine and the algorithms formally using the I/O Au- 
tomata (IOA)[20] and Timed IOA [12,11] models, in fact, using formal spec- 
ification languages that have been defined for these models. The IOA/TIOA 
specification languages lead to very stylized assertional proofs for invariants and 
simulation relations that can be partially automated using theorem provers. 
Moreover, the same IOA specifications can be used by the IOA compiler [31, 30] 
to produce executable Java code. 


Other related work: Our use of a partial order automaton as an abstract spec- 
ification was inspired by prior work of Fekete et al. on specifying the behavior 
of an Eventually Serializable Data Service [9]. Their specification used a (dif- 
ferent) partial-order machine, which expresses weaker consistency requirements 
than atomicity. The algorithm studied in [9], based on an earlier algorithm of 
Liskov et al. [13], was shown to achieve this weaker form of consistency. 

The only other published simulation-based atomicity proofs we are aware of 
are those of Bogdanov [5] (replicated state machine), and Doherty et al. (lock- 
free queue) [7]. The proofs in both these papers are complicated: They involve 
multiple levels of asbtraction as well as both forward and backward simulations. 
In contrast, every construction considered in this paper is shown to be atomic 
by exhibiting a single forward simulation directly from the implementation au- 
tomaton to a specification automaton. 

Another example of using assertional reasoning for proving atomicity is the 
work by Wang and Stoller [37], which uses static analysis combined with model 
checking to verify atomicity of code blocks involving lock-free synchronization 
primitives. A more general discussion of assertional proof techniques can be 
found in [28]. 


The rest of the paper is organized as follows: In Section 2, we introduce 
preliminary definitions and notation used throughout the paper. The sufficient 
condition for proving atomicity is specified in Section 3. The PO-Machine is 
described in Section 4. The ABD algorithm is presented and proved correct in 
Section 5. A time-based version of ABD is discussed in Section 6. Section 7 
briefly discusses the proofs of the Vitanyi-Awerbuch’s register construction, and 
of the Lamport’s replicated state machine. Section 8 discusses future directions. 
For lack of space, we only outline intuition and highlight basic ideas underlying 
the correctness proofs. The detailed proofs can be found in the full version of 
the paper [6]. 


2 Preliminary Definitions 


We use the I/O Automata (IOA)[20] model to formally specify services, describe 
algorithms and carry out proofs. An I/O automaton is a non-determenistic state 
machine whose state can change atomically through a discrete transition labeled 
by a discrete action. The set of the automaton’s actions is called the action 
signature of the automaton. The actions can be either external or internal. The 
external actions, which can be either input or output, model interaction with 
the automaton’s environment; and the internal actions model local computation 
steps. In Section 6, we also use the Timed I/O Automata (TIOA) model [12, 
11], which, in addition to discrete transitions, also allows the automata state to 
evolve by trajectories, which describe evolution of the state over time. 

We use forward simulations to carry out atomicity proofs. Informally, a for- 
ward simulation is a relationship between the states of two automata requiring 
that the transitions of one system can in some sense be mimicked by the other. 
A precise definition of the simulation formalism can be found in [21]. 


The read/write service A read/write object (a register) type consists of the 
following components: (1) an arbitrary set of values V with an initial value vp, 
(2) the set of operations of the form write(v), v € V, and read, (3) the set of 
responses are ack and v € V, and (4) the sequential specification f such that 
f(w, write(v)) = (v, ack) and f(w, read) = (w,w). 

A read/write service implements a shared read/write register. To access the 
service, a client issues an operation descriptor consisting of a location identifier 
loc, and an operation identifier id. In addition, the write operation descriptor 
also contains a value val. We often refer to an operations descriptor x simply 
as operation x, and denote its various components by x.loc, x.id, and x.val. We 
denote by O,, and O,. the sets of the write and the read operations respectively, 
and by O = O,, UO, the set of all operations. For a set X C O, we denote by 
X.id = {x.id: x € X} the set of identifiers of operations in X. 

Clients use the actions of the form request(x), x € O, and response(z,v), 7 € 
O,v € VU {ack}, to issue operation requests and receive responses respectively. 
Given a sequence (7 of the request and response actions, an requested operation 
x is said to be complete in ( if 6 contains response(x,v) for some v € V U {ack} 
which we call the return value of x. 


We say that ( is well-formed if there exists a function cause mapping each 
response event to a preceding request event in 3 so that the following is satisfied: 
(1) For each response event e = response(x,*), cause(e) = request(x) (i.e., re- 
sponses are not spuriously generated); and (2) cause is one-to-one (i.e., responses 
are not duplicated). 

The following definition will be used throughout the paper: Let IT be a set of 
read and write operations, and R be a binary relation over I7. For an operation 
mw € IT we define last-prec-writes(7,R) = {w € Oy: (wim) ECE RA Ae’ €O,: 
(w,w’) E RA (w’,7) € R}. 


3 Atomicity 


Atomicity (or linearizability) is specified as a property satisfied by the object 
implementation traces. It is typically defined in terms of the existence of serial- 
ization points for operations so that shrinking the operations to occur at their 
serialization points results in a valid sequential execution of the read/write reg- 
ister (see, e.g., Chapter 13 of [21], Chapter 9 of [4], or [10]). For our purposes in 
this paper, it is enough to give a sufficient condition for proving atomicity; this 
condition is equivalent to the one in Lemma 13.16 of [21]. 

Let @ be a well-formed sequence of the actions of the read/write service 
interface that contains no incomplete operations, and IT be the set of opera- 
tions requested in 3. We say that ( satisfies Partial Order property (henceforth, 
referred to as PO) if there exists an irreflexive partial ordering ~ of all the 
operations in J, satisfying the following: 


Property 1 (PO Constraints) 


1. If the response event for x precedes the request event for d in G, then d & a. 

2. For any two write operations n and @ in II, either m =~ @ or Ox 7. 

&. If x is a write operation in IT and ¢ is a read operation in IT whose request 
event follows the response event for 7, then 7 ~ ¢. 

4. If m is a read operation in IT and ¢ is a read operation whose request event 
follows the response event for 7, then for each w € last-prec-writes(m, ~), 
w~< ¢. 

5. Let m be a read operation in IT, and v be the value returned by m. If 
last-prec-writes(7,<) #0, then v =w.val for some 
w € last-prec-writes(7,<). Otherwise, v = vo. 


The following lemma is proved in [6]: 


Lemma 1. @ satisfies PO iff there exists an irreflexive partial ordering of all 
the operations in IT, satisfying the (more restrictive) constraints of Lemma 18.16 


of [21]. 
From the above result and Lemma 13.16 of [21], we obtain: 
Lemma 2. /f (3 is well-formed and satisfies PO, then 2 satisfies atomicity. 


' Note that our notion of well formedness is weaker than that usually found in the 
literature as it allows requests from the same location to be issued concurrently. 


4 The PO-Machine 


In this section we define the Partial-Order Machine. First, we formally specify 
the environment assumptions of the read/write service. This environment is 
represented by a single automaton, called Users, whose code could be found 
in [6]. The Users automaton contains a single variable requested to keep track 
of the ids of requested operations, in order to avoid repeats. An implementation 
of the environment would not have such a variable, but would use some other 
mechanism to ensure unique operation ids (e.g., client id and a counter). 


Lemma 3. For x,y € requested, x =y @ vid = y.id. 


The PO-Machine signature and state variables appear in Figure 1, and its 
transitions appear in Figure 2. This automaton maintains a partial order in 
its state, represented by variables vertices and edges. Vertices correspond to 
requested operations, and edges to ordering relationships that have been deter- 
mined for these operations. When a request arrives, it is put into vertices; later, 
it becomes classified as ordered, then completed, and finally, responded. Edges 
may be added at any time from ordered write operations to unordered ones (see 
action add-edge). 

An unordered operation 7 may become ordered at any time after it has 
acquired incoming edges from all write operations that completed before 7 began 
(i.e., all writes in prec(7)). This ensures that constraints 1 and 3 of Property 1 
hold among all writes, and between writes and reads. Constraint 1 is also trivially 
preserved among reads as edges originating at read requests are disallowed by 
the PO signature (see Figure 1). When a write operation 7 becomes ordered, 
new edges are inserted to ensure that 7 is ordered with respect to all previously- 
ordered write operations (see action order) so that constraint 2 of Property 1 is 
satisfied. 

An ordered operation may become completed at any time; when a read oper- 
ation ¢ completes, it also forces each write operation 7 immediately preceding ¢ 
in the partial order to complete. This ensures that every read operation invoked 
after ¢ completes will find 7 in its prec set, and will therefore, become ordered 
only after it has an incoming edge from 7. This guarantees that constraint 4 of 
Property 1 is satisfied, and also captures the essence of the “helping” mechanism 
found in many atomic register implementations. 

A completed operation is allowed to return a response. The response returned 
by a read operation is the value written by the last preceding (in the partial 
order) write operation, or the initial value if no such write exists (see action 
response). Thus, constraint 5 of Property 1 is satisfied. 

In [6], we prove that the limit of the transitive closure of (vertices, edges), 
maintained in the derived variable dag, satisfies Property 1. Since every trace 
of PO-Machine is obviously well-formed, by Lemma 2, PO-Machine implements 
an atomic register: 


Theorem 1. Each trace of the PO-Machine satisfies atomicity. 


Signature: 


Input: Output: Internal: 
request(z), x2 € O response(z,v), x € O, add-edge(x, y), 7 € Ow, y € Ow UOr 
v €VU {ack} order(x), « € O 


complete(x), « € O 


State: 

vertices C O, initially empty responded C O, initially empty 

ordered C O, initially empty edges C O x O, initially empty 

completed C O, initially empty prec is a partial function from © to subsets of O, 


initially empty 


Derived vars: 
dag, the transitive closure of (vertices, edges) 


For x € Op, last-writes(x) = last-prec-writes(a, dag) 


Fig. 1. PO-Machine signature and states 


Input request(«) Internal complete(x) 
Effect: Precondition: 
vertices := vertices U {x} «x € ordered — completed 
prec(x) := completed N Ow Effect: 
completed := completed U {x} 
if x € Op then 
Internal add-edge(x, y) Vy € last-writes(a) do 
Precondition: completed := completed U {y} 
y € vertices — ordered 
x € ordered 
Effect: Output response(xz, ack), x € Ow 
edges := edges U {(a, y)} Precondition: 
x € completed — responded 
Effect: 
Internal order(x), az € Ow responded := responded U {x} 
Precondition: 
x € vertices — ordered 
Vy € prec(x) : (y,x) € dag Output response(x, vg), © € Or 
Effect: Precondition: 
edges := edges U {(a, y): y € ordered N Ow A x € completed — responded 
(y, x) ¢ dag} last-writes(x) = 0 
ordered := ordered U {x} Effect: 


responded := responded U {x} 


Internal order(x), 7 € Or 


Precondition: Output response(x, v), 2 € Or 
«x € vertices — ordered Precondition: 
Vy € prec(x) : (y,x) € dag x € completed — responded 
Effect: last-writes(x) A 0 
ordered := ordered U {x} v= w.val : w € last-writes(x) 
Effect: 


responded := responded U {x} 


Fig. 2. PO-Machine transitions 


5 The Attiya, Bar-Noy, and Dolev Algorithm 


In this section, we present a distributed wait-free implementation of an atomic 
multi-writer/multi-reader register based on the well-known message-passing al- 
gorithm of Attiya, Bar-Noy, and Dolev [3] (which we call ABD). We prove cor- 
rectness of ABD by showing that ABD implements PO-Machine, which by The- 
orem 1, implies that ABD implements an atomic register. 

The original ABD protocol implements a wait-free atomic read/write regis- 
ter using a collection of n processes communicating among themselves through 
reliable point-to-point channels. The implementation is resilient to up to n/2 pro- 
cess crashes. Each process in ABD is responsible for both: handling the client 
operation requests, and storing and updating the local copy of the register value. 


Here, we present a generalized version of ABD where we let the two roles in 
the ABD protocol be performed by two classes of agents: clients and replicas. 
This design allows for flexibility in assigning roles to actual network locations 
thus simplifying the algorithm deployment in real systems. We also use a sepa- 
rate client to handle each user request so that the actual clients can handle any 
number of requests and in whatever order (for example, requests can be par- 
titioned among several threads, or executed sequentially). Our implementation 
also supports multiple writers using the technique of [23]. 

We now describe the ABD implementation (the ABD automaton) in more 
detail. Let P be a finite set of replicas. We define a quorum system Q on P to 
be the union of a set of write quorums Q, and the set of read quorums Q,. 
O,, and Q,. are sets of subsets of P such that for each Qy € Q, and Q; € Q,, 
Qw OQ, #0. The ABD automaton is the composition of the Users automaton 
of Section 4, the client automata C,, x € O, the replica automata R,, p € P, 
and the reliable point-to-point channel automata connecting each client C, with 
replica R, and vice versa. The client’s interface and state variables appear in 
Figure 3. The code of the reader client, the writer client and the replica appear 
in Figures 4, 5, and 6 respectively. We do not present the specification for the 
channel automata as their functionality is obvious. 

The value stored at each replica is associated with a tag. Tags are two-field 
records consisting of a sequence number sn, which is a non-negative integer, and 
a request identifier 7d. Tags are ordered lexicographically with the precendence 
to the sequence number field. 

Clients access read (resp. write) quorums by first sending a message to all 
the replicas, and then awaiting responses from a write (resp. a read) quorum. 
The request handling at clients involves two rounds of quorum accesses, called 
the read phase and the write phase respectively, such that a read quorum is 
contacted during the read phase, and a write quorum is contacted during the 
write phase. A client keeps track of the request progress through the phases 
using the variable status. The operation’s status is initially idle. It is changed 
to pending (p) at the beginning of the read phase. It becomes sending (s) at the 
beginning of the write phase. It is changed to committed (c) upon completion of 
the write phase, and finally to responded (r) after a response is returned. 

Specifically, to handle a write request x, the client C, (see Figure 5) performs 
a read phase to determine the highest tag ¢ associated with the values stored at 
some read quorum. It then performs a write phase to store the value v associated 
with tag (t.sn,z.id) at a write quorum. It then responds with ack. To handle a 
read request y, client C, (see Figure 4) first performs a read phase to determine 
the value v associated with the highest tag t among those associated with the 
values stored at some read quorum. It then performs a write phase to guarantee 
that the pair (t,v) is stored at a write quorum. It then responds with v. 

The replica’s algorithm (see Figure 6) is simple: In response to a read phase 
message, a replica p either responds with its current tag (for write requests), or 
the current tag and the value (for read requests). In response to a write phase 
message carrying a tag which is bigger than p’s current tag, p overwrites its 


current tag and the value with those in the message. Otherwise, the p’s state is 
left unchanged. In both cases, p responds with ack. 


Types: 
Tag = N2° x ©.id, with selectors sn and id, ordered lexicographically 


Phase = (idle, p,s,c,r}, ordered so that idle << p<s<c<r 


Signature: 
Input: Output: 

request (x) response(a, vu), uv © V U {ack} 

receive(m)p,2, p € P, m € {ack} U send(m)a,p, p © P, me {r,w} U 

N20 U (Tag x V) (Tag x V) 

Internal: 

rq-collected(q)z, g@ € Or 

wq-collected(q)a2, gq € Qw 
State: 
status € Phase, initially idle read-resp € P, initially empty 
val € V, initially undefined write-resp € P, initilly empty 
tag € Tag, initially (0, ig) for each p € P: req-buffer, € seqof({r, w} U 

(Tag X V)), initially » 
Fig. 3. The state and signature of client automata C,, x € O for ABD. 
Input request(«) Input receive(ack)p, x 
Effect: Effect: 
status :=p write-resp := write-resp U {p} 


for each p € P: 


append (r) to req-buffer 
{r) P Internal wq-collected(q) x 


Precondition: 
Input receive(v, t)p,x status =s 
Effect: write-resp D q 
read-resp := read-resp U {p} Effect: 
if status =pAt > tag then status :=c¢ 
val:= vu 
fg eete Output response(a, v) 
Precondition: 
Internal rq-collected(q) x status =c 
Precondition: val=v 
status =p Effect: 
read-resp D q status :=r 
Effect: 
status :=s 


Output send(m)x,p 
Precondition: 

req-buffer p # 

m = head(req-bufferp) 
Effect: 

delete head of req-bufferp 


for each p € P: 
append (tag, val) to req-buffer py 


Fig. 4. Transitions of reader Cz, « € O, for ABD. 


Correctness of ABD: We now prove that ABD implements an atomic register. 
Our strategy will be to show that ABD implements PO-Machine by exhibiting a 
forward simulation from ABD to PO-Machine. In the following, for each x € O, 
we will use subscript x to refer to the state variables of C,. It is convenient 
for the ABD correctness proof to define several derived variables for the ABD 
automaton. These are summarized in Figure 7. 


Input request(«) Input receive(ack)p, x 

Effect: Effect: 
status :=p write-resp := write-resp U {p} 
for each p € P: 


append (w) to req-buffer. 
PP: (w) buff P Internal wq-collected(q) x 


Precondition: 
Input receive(sn)p,r, sn € N29 status =s 
Effect: write-resp D q 
read-resp := read-resp U {p} Effect: 
if status =p A sn > tag.sn then status :=c¢ 
tag.sn := sn 
Output response(x, ack) 
Internal rq-collected(q) x Precondition: 
Precondition: status =c 
status =p 
read-resp D q Effect: 
Effect: status :=r 
status :=s 
tag.sn := tag.sn+1 Output send(m)a,p 
foneech EG Fs Precondition: 
append (tag, x.val) to req-bufferp i. head eau) 
Effect: 


delete head of req-bufferp 


Fig. 5. Transitions of writer Cz, x € Ow for ABD. 


Signature: 
Input: Output: 

receive(m)x,p, « € O, me {r,w}U (Tag x V) send(m)p,2, © € O,p € R, me {ack} U (Tag x V) 
State: 


val € V, initially vg 
tag € Tag, initially (0, ig) 


For each « € O: resp-bufferg, € seqof({ack} UN2® U (Tag x V)), initially » 


Transitions: 
Input receive(r) x ,p Input receive(t, v)x,p Output send(m)p,ax 
Effect: Effect: Precondition: 
append (val, tag) to resp-buffer x if t > tag then resp-buffer, # 
tag:=t m = head(resp-bufferg ) 
; val :=v Effect: 
Input receive(w) a ,p append (ack) to resp-buffery, delete head of resp-bufferz 


Effect: 
append (tag.sn) to resp-buffer; 


Fig. 6. Replica automaton R,, p € P for ABD 


Among these variables, the most interesting one is min-tag which is used to 
keep track of the lowest possible tag that could ever be determined by a client at 
the end of the read phase. At the beginning and before any replica has responded, 
min-tag is the smallest tag among the maximum tags carried by replicas in every 
read quorum. As the client is progressing through the read phase it might get a 
response from a replica whose tag is bigger than the current value of min-tag. 
In this case, the definition of min-tag ensures that min-tag is assigned to that 
higher value. Finally, upon completion of the read phase, the value of min-tag 
is fixed to be the maximum tag received during the phase. The simulation proof 
relies on the following key property of min-tag: 


Lemma 4. For each x € O, min-tag(x) is non-decreasing. 


The simulation mapping from the states of ABD to the states of the PO- 
Machine appears in Figure 8. The first four components of the mapping are 


pending = {x € O: statusg > p} 

ordered = {x € O: status, > s} 

completed = {x € O: statusy, > c} 

responded = {x € O: statusy, > r} 

For r € Or: last-writes(r) = {w € Ow M ordered: s.tagw = s.tagr} 
For « € O, p € P: 


t, if Su © V: (vu, t) € resp-bufferp,., U channelp,x 
new-tag(«,p) = 4 (sn,a.id), if (sn) € resp-bufferp,, U channelp,x 
tagp, otherwise 


— Forz€O: 
max(tagz,mingeg,, max{new-tag(x, p) : p € Q \ read-respy }], 
min-tag(x) = if VQ € Qp, read-resp BD Q 


tagx, otherwise 


Fig. 7. Derived variables for the ABD automaton 


straightforward: All the operations that have ever been requested (indicated by 
status > idle) are mapped to vertices; the operations that have completed the 
read phase and acquired final tags (indicated by status > p) are mapped to 
ordered; and the operations that have responded (indicated by status > c) are 
mapped to responded. 

The set of edges consists only of edges among operations that have completed 
their read phases (8.7). The edges among these operations are determined by 
their tag order and type. Specifically, any two writes x and y, such that tag, < 
tagy, are connected by edge (x,y) (8.8); and each read a and write y such 
that tag, = tag,, are connected through edge (y,x) (8.9). To maintain the 
mapping for edges, each rq-collected(a) for « € O., is simulated by a sequence of 
add-edge(y, «) for each ordered write operation y such that tag, < tag,, followed 
by order(a); and each rq-collected(x) for « € Oy is simulated by a sequence of 
add-edge(y, x) for each ordered operation y such that tag, = tag,. No actions 
involving unordered operations (i.e., the operations with status < s) result in 
adding new edges. 


f is the relation over states(PO — Machine) x states(ABD) such that each (s,u) € f iff: 


uiréequestéd = s.reduested 
uvertices-ccpending 
wrordenéd’ = s.ondered 
u:completed = s.completed U U a. last-writes(1r) 
r€OrNs.completed 
uiresponded =:Siresponded 
For all x € u.vertices, if y € u-prec(x), then s.tagy < s.min-tag(x) 
u.dag C s.ordered x s.ordered 
For all 2, y € Ow M w.ordered, if (a, y) € u.dag, then s.tage < s.tagy 
For all 2 € Ow Nu.ordered and y € Oy Nu.ordered, (x,y) € w.edges iff s.tagx = s.tagy 


SOD? DOW LR Connors 


Fig. 8. Forward simulation from ABD to PO-Machine 


The most interesting part of the proof is to show that order(a) becomes 
enabled once all the (y,2) edges have been added. For that we need to show 
that the tag acquired by x at the end of the read phase is at least as big as 
the tag of every operation that had completed before x began. Since at the 


end of the read phase, tag, = min-tag(x), the necessary enabling condition is 
provided by part 8.6 of the mapping that requires that for each y € prec(z), 
tagy < min-tag(x). 

To show that 8.6 is maintained throughout the read phase of x, request(x) 
is simulated by the request(x) action of the PO-Machine; and each receive is 
simulated by the empty sequence. Since at the time x is invoked, the tag of every 
y € prec(x) has been stored at a write quorum of replicas, and because every pair 
of write and read quorums intersects, mingeg, Maxpeg{tagp} > tagy. Hence, 8.6 
is preserved by request(«). Finally, since min-tag(x) is non-decreasing (Lemma 4) 
and prec(x) is not affected by any action except request, 8.6 is preserved by 
receive. Hence, by the end of the read phase of x, for each y € prec(x), tagy < 
min-tag(a) as required. 

We argued informally that the mapping in Figure 8 is a forward simulation 
from ABD to the PO-Machine. A detailed proof appears in [6]. 


Lemma 5. The mapping in Figure 8 is a forward simulation from ABD to the 
PO-Machine. 


Since by Theorem 1, each trace of the PO-Machine satisfies atomicity, the 
same is true for every trace of ABD: 


Theorem 2. Each trace of ABD satisfies atomicity. 


Automated Tools Support: We have used the TIOA to PVS translator and TAME 
library [2] to generate descriptions of the PO-Machine and the ABD algorithm 
in the language of the Prototype Verification System (PVS) [26]. We used PVS 
to substantially increase the level of detail and assurance of some of our previous 
hand proofs. In fact, we discovered several gaps and bugs in our hand proofs. 
Automatic translation enabled us to easily tweak the simulation relations and 
rerun the proof scripts. We also used the IOA code generator tool [31,30] to 
compile the verified ABD automaton into an executable Java code. This way, 
a single formal representation of the ABD algorithm was used for specification, 
verification, and execution. 


6 Timed ABD 


In this section, we present an optimized version of the ABD protocol, called 
Timed-ABD, that takes advantage of perfectly synchronized clocks at the writers 
to eliminate the read phase of the write implementation (see [8]). 

The Timed-ABD is the composition of the following timed automata: the 
replica and reader client automata in Figures 6 and 4 respectively augmented 
with arbitrary trajectories that keep their state unchanged; and the writer client 
automata whose code appears in Figure 9. To model synchronized clocks, each 
writer maintains a local variable clock whose trajectory is d(clock) = 1 (i.e., the 
clock value grows continuously, at the same rate as the real time). 

The writer algorithm is as follows: To write a value, the writer first takes its 
current clock reading, and then delays its execution until its clock exceeds the 


Signature: 


Input: Internal: Output: 

request (a) ; order, response(x, vu), vu © {ack} 

receive(™m) pn, P € P, m € {ack} UN2zO wq-collected(q)r, g © Qw send(m)a,p, m € {w} U (Tag x V) 
State: 


clock € R, initially 0 
Discrete req-time € R, initially 0 
status € Phase, initially idle 


Transitions: 


Input request(a) 


Effect: 
status :=p 
req-time := clock 


tag € Tag, initially (0, x.id) 

write-resp C P, initilly empty 

for each p € P: req-buffery € seqof({w} U (Tag x V)), 
initially 


Input receive(ack) p, x 
Effect: 
write-resp := write-resp U {p} 


Internal wq-collected(q) x 


Internal orderz, Precondition: 
Precondition: status =s 
clock > req-time write-resp D q 
status =p Effect: 
Effect: status :=c 
tag.sn := clock 
status :=s 
for each p € P: Output response(x, ack) 
append (tag, x.val) to req-bufferp Precondition; 
status =c 
Effect: 
status :=r 
Trajectories: 
evolve Output send(m)a,p 
Precondition: 
d(clock) = 1 m = head(req-bufferp ) 
Effect: 


All the other state variables are kept unchanged delete head of req-buffer-p 


Fig. 9. Writer client Cz, « € Ow for Timed-ABD 


initial reading. The second clock reading is used as the tag with which the client 
performs the write phase. 


The simulation mapping from the states of Timed-ABD to the states of 
Timed-PO (i.e., the PO-Machine augmented with arbitrary trajectories that do 
not change its state) appears in Figure 10. To see that the mapping is preserved, 
we observe that a write operation becomes ordered once it is verified that a 
non-zero amount of time has elapsed since it was requested. We therefore, simu- 
late each Timed-ABD trajectory corresponding to a non-zero time interval by a 
trajectory of Timed-PO of the same length, followed by a sequence of add-edge 
actions, followed by order. The rest of the simulation proof is straightforward 
(see [6] for details). 


f is the relation over states(Timed-ABD) X states(Timed-PO) such that (s,u) € f iff: 


Identical to 1-5 in Figure 8 
For all « € u.vertices M Op, if y € u.prec(«), then s.tagy < s.min-tag(x) 

Identical to 7-8 in Figure 8 

For all « € u.verticesN Ow, if y € u.prec(«), then s.tagy.sn < s.req-timey 

For all « € (u.vertices — u.ordered) N Ow, y € u.ordered N Ow, if s.tagy.sn < s.clocky, then (y,x) € 
u.edges. 


my 
O02 


Fig. 10. Forward simulation from Timed-ABD to Timed-PO 


7 Other Algorithms 


We discuss briefly how to prove atomicity of the unbounded multi-writer /multi- 
reader register construction of Vitanyi and Awerbuch [36] (referred to henceforth 
as VA), and of a general atomic object implementation based on the replicated 
state machine algorithm of Lamport [15] (referred to henceforth as RSM). 

First, we observe that VA can be recast as a special case of ABD with the 
write quorums being the rows and the read quorums being the collumns of the 
matrix. Therefore, the simulation proof of VA is almost identical to that of ABD. 
In particular, it is easy to see that the simulation from ABD to PO-Machine in 
Figure 8 is also a forward simulation from VA to PO-Machine. 

To prove atomicity of RSM, we use a simplified version of the PO-Machine, 
called TO-Machine. The TO-Machine constructs a single total order of all the 
requested operations. In particular, every operation becomes ordered only after 
it is ordered relative to all the other ordered operations. The TO-Machine is 
parameterized by the emulated object sequential specification and initial state 
which are used to compute responses. The simulation proof is based on the ob- 
servation that in RSM, an operation x becomes ordered once the local timestamp 
at each replica becomes greater than that of «. The full proof appears in [6]. 


8 Conclusions and Future Work 


Our work with four algorithms so far suggests to us that our PO-Machine (or 
small variants) may be general enough to capture many of the existing atomic 
register algorithms. We plan to use these methods to study a wider variety of al- 
gorithms, such as bounded-timestamp-based constructions (see e.g., [34]), whose 
proofs have been notoriously difficult and bug-prone. An interesting challenge 
will be to extend our framework to capture implementations that are not ex- 
plicitly based on timestamps, for example, the construction that creates atomic 
bits from safe bits [82]. Another interesting direction deals with adapting the 
PO-Machine to capture weaker register semantics, such as safe registers, reg- 
ular registers (including the multi-writer regular registers of Welch [29]), and 
sequentially consistent registers. There is an increased recent interest in these 
semantics as they capture the guarantees provided by many Byzantine-resilient 
storage systems [24, 25,1] based on Byzantine quorums [24]. 

Yet another interesting application domain for our techniques is the verifi- 
cation of multi-threaded programs based on lock-free synchronization primitives 
(such as CAS, LL/SC, etc.). This area has recently been receiving an increased 
attention due to the growing popularity of multi-processor computing platforms, 
and the introduction of lock-free synchronization primitives into the Java con- 
currency package. 

Finally, we are interested in identifying common patterns behind many di- 
verse implementations of atomic objects. This will make it easier to understand 
and compare different algorithms. We expect that such patterns should be ex- 
pressible in terms of common specification automata (e.g., a unified version of 
the PO- and TO-Machines). 
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